Skip to main content

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Jonathan Aizen Date: May 4, 2003 5:04am
Forum: etree Subject: Re: Putting email addresses in the details pages

The /audio part of this site isn't crawled because it's excluded in the robots.txt - I think this is a remnant and should be changed. If changed, the section will be heavily crawled by Google - they currently crawl the forums and other parts of the site multiple times per day.

Info files (txt files on the etree0x servers) are actually crawled already because those machines don't have robots exclusions.

Reply to this post
Reply [edit]

Poster: glenn Date: May 7, 2003 1:29pm
Forum: etree Subject: Re: Putting email addresses in the details pages

there do exist user-agents which ignore the robots.txt exclusion file, so it is safe to assume that all portions of this site can be crawled by malicious (email-address-harvesting) user-agents.
A better place to put seeder's email addresses, in my opinion, is in the info.txt file included with the seed.

Reply to this post
Reply [edit]

Poster: Jonathan Aizen Date: May 7, 2003 2:24pm
Forum: etree Subject: Re: Putting email addresses in the details pages

Again, the text files are crawled, so that won't help...